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Abstract 

This paper describes a study of success factors in the introductory semester of liberal studies 
blended courses offered at the bachelor of science level. The influence of student participation in 
the online course environment was examined, as measured by the number of times students 
logged into the learning management system (LMS) and average session length. These measures 
were correlated with final course grades to increase understanding of the participation patterns of 
successful students. The resulting patterns and their implications are identified. We observe that 
students with an intermediate number of logins and average session length tended to exhibit the 
optimal level of course performance with students who logged in near the low or high amount of 
times tending to receive lower grades. 
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Introduction 

In an effort to provide students of a women’s college in the Middle East with the highest quality 
learning experiences, flexible access to courses, and most accessible channels of communication 
among students and faculty, staff examined the potential for expanded use of open source software 
and platforms. The intent of the activity was to identify a rigorous and extensible system for 
teachers to build and share dynamic learning objects with students and between faculty. The pool 
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of candidates for possible open Course or Learning Management Systems (LMSs) included Sakai 
and Moodle. Although these systems provided similar options, the college choose to pilot Sakai in 
an attempt to determine feasibility of use for blended courses, with the possibility of disseminating 
content and participation to other colleges in the region. Sakai has a modified approach to the 
traditional LMS, promoting the tool as a sharing or collaborative environment, hence the label of 
Collaborative Learning Environment (CLE). Sakai offers the core functions of prior LMSs with the 
addition of being open, which some interpret to mean free. Although there is no licensing for the 
software, the cost is in the human and technical resources required by hosting, maintenance, and 
programming. The college chose to outsource these responsibilities to a reputable hosting 
company in Europe, due to their proximity of time zone. The fee to host a two semester project for 
the college that enrolled 2,500 students on two campuses was $10,000 USD, which included all 
services as well as sufficient template modification to offer a branded site for the college. Both 
parties were aware that the cost was reduced as the intent was to produce proof of concept data, to 
enable full integration in subsequent years within the college, and potentially to expand use of the 
CLE throughout the multi-campus system. 

The CLE hosting business’s leaders and college staff worked via campus meeting and Web 
meetings, led by the host Project Manager and the college Technical Services Department 
Supervisor. The host provided training to the college technical staff, who were already experienced 
in managing LMS platforms. The goal of the pilot project was to create an effective and flexible 
system for student engagement, as well as instructor sharing of their resources, lessons, 
assessments, and so on. The anticipated outcome was that this system would be as good as, if not 
better than, the current LMS used in the college, which required a substantial licensing fee. The 
approach adopted for the pilot project was a quasi-experimental design in which both LMS 
platforms would be used in parallel for sufficient numbers of sections, instructors, and students to 
generalize the findings into other courses and campuses. Ultimately, the college found that Sakai 
could offer more than the commercial LMS, at a significantly reduced cost, and with no usability 
issues for faculty or students. 

This paper describes key findings from the open CLE pilot program. This study focused on success 
factors in the introductory semester of liberal studies blended courses offered at the bachelor of 
science level. The influence of student participation in the online course environment was 
examined, as measured by the number of times students logged into the learning management 
system (LMS) and average session length. These measures were correlated with final course 
grades in order to develop recommendations for blended course design and instruction, which 
must apply evidence for how students learn to the micro and macro levels of content chunking 
(Keeler, 2015). This study addressed the blended learning success factor of iterative and recursive 
design (Ferdig, Cavanaugh, & Freidhoff, 2015), and provides a model for programs that seek to 
refine their blended course design approach. 

Blended Learning for Liberal Studies 

Tertiary education is increasingly being redefined from site to service (Cavanaugh & Hargis, 2010) 
as the emphasis in education shifts from being a location where students have similar experiences 
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along similar timelines depending on system needs to being a personalized program provided as 
needed, where needed, and when needed depending on student needs. This site-to-service shift 
includes several changes that we see emerging as influences on the design and delivery of 
education, as shown in Table 1. 

Table 1 

Shifts from Site to Service 


Shifts 

Site: campus 

Service: education 

Place 

Classroom for lectures and 
assessments; home for practice 

Campus, home, community, and field for 
all activities 

Time 

6.5hours/day; 180 days/year between 
September and June 

Any time, any day, year-round 

Materials 

Physical and virtual; owned by the 
institution with limited access off 
campus 

Physical and virtual; increasingly in 
public domain and by student 
subscription for access anytime on 
student devices 

Interactions 

Teacher-directed; mostly occurring in 
class 

Teacher-moderated; occurring anytime 

Assessments 

Scheduled for all students according 
to syllabus 

Given as individual students reach 
mastery 


These shifts in the design and delivery of formal education have so far been largely made possible 
by technology, and their degree of priority has increased due to a range of economic and social 
conditions. Colleges are serving a wider range of students in degree programs, are under pressure 
to make efficient use of space and resources, are held accountable for each student’s academic 
performance, are seen as key drivers in the nation’s economic competitiveness, and thus, are 
depended upon to provide career-ready graduates. Because the technology now exists for 
differentiated instruction, these priorities are increasingly met through blended education 
programs that merge physical and virtual learning environments to serve students with greater 
flexibility and agility. This article describes a study of a semester-long college pilot program in 
which blended undergraduate courses were offered and data were analyzed to identify factors 
related to academic success in the courses. A blended approach to course design was especially 
appealing for the pre-professional students in the college who spent increasing time in 
professional settings during their baccalaureate degree programs, and could benefit from the 
added time flexibility afforded by blended course delivery. 

In tertiary education, examples of technology-enabled blended education date from the beginning 
of educational technology. Classroom learning time has been enhanced, augmented, and partially 
replaced by technology for as long as the technology has been available (Ferdig & Kennedy, 2014). 
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The past decade has seen a steady increase in the development and implementation of blended 
education programs as technology has become more powerful and ubiquitous. However, it has 
been only in the past few years that deliberate blended education programs have developed (Clark 
& Barbour, 2015). This progress has been spurred in large part by open source learning 
management systems and more readily available education resources. 

In the Middle East and North Africa (MENA) region where the college participating in this study is 
located, open education resources are expanding, as shown in a recent survey in the United Arab 
Emirates (Khelifi, Abu Talib, A 1 Shabibi, A 1 Zaabi, & A 1 Marshoudi, 2010). Open source software 
and its associated benefits in terms of cost efficiency, flexibility, and participant content creation 
are regarded as positive supports for scaling up access to education in countries with 
comparatively new education systems: “an open learning environment and course should offer 
youth an opportunity to personalize their learning, to make it meaningful, authentic, and engaging. 
Open learning creates the opportunity to offer all networks a chance to connect and learn together 
regardless of age, experience, culture, or background” (Graham, Labonte, Roberts, O’Byrne, & 
Osterhout, 2014, p. 460). 

Effective blended education approaches in higher education have been documented (Bonk & 
Graham, 2005; Garrison & Vaughan, 2008; Picciano & Dziuban, 2010), but none have centered in 
the Persian Gulf region and few report data on student performance. Those that have done so 
suggest that blended programs can contribute to learning gains (US Department of Education, 
2009; Laumakis, Graham, & Dzubian, 2009). Further, very little research examines specific 
factors within blended courses and environments that correspond to student academic 
performance. The college program profiled here fills that gap by analyzing participation of all 
students enrolled in a semester of blended courses that used the Sakai CLE to enhance and expand 
classroom learning. 

While each blended course has unique features that intersect in different ways with different types 
of students, findings of research on the characteristics of successful blended learners suggest a 
common set of characteristics likely to result in successful virtual learning. The expanded learning 
time afforded by a flexible blended course timeline has shown success in raising student 
achievement (Cavanaugh, 2009). Indeed, time was identified as the most significant factor in 
student success in 11 of 15 high enrollment blended courses (Liu & Cavanaugh, 2011). This finding 
encourages the design of courses to engage and motivate students to spend more time in the 
system. Specifically, Liu and Cavanaugh (2011) observed that the frequency with which students 
accessed online materials related to achievement in ways that seem to depend on the complexity of 
the course, in that more logins corresponded to higher achievement in introductory level courses, 
while fewer logins for longer sessions corresponded to higher achievement in advanced courses. 
These findings guided the college open source blended course data analysis in spring of 2012. 

The research question guiding this study is “Does student level of participation in the online 
learning environment correspond to student academic performance in a blended college course? If 
so, how?” 
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Methods 


Setting and Participants 

Data were collected during a full semester when the blended course design was introduced in a 
mid-sized four-year women’s college in the United Arab Emirates. During this semester, 590 
students and 14 instructors were registered in course sections created in Sakai CLE for courses 
that included classroom and online experiences. Seven courses were designed in-house for the 
CLE. Of these courses, one was developed and taught in a way that integrated physical and virtual 
interactions. This course (LSS 2533—see Table 2) is part of the bachelor-level liberal studies 
curriculum, and was taught by seven experienced faculty members for a total of 477 students using 
the same instructional materials, assignments, and assessments as the fully face-to-face versions 
of the course. Students were registered in LSS 2533 according to their program plans and 
schedules. Three hundred twenty-one students completed the course and were included in this 
study. Because the college serves only female National citizens of the countiy where it operates, 
students in the study sample represent a narrow ethnic and socioeconomic range in one large city. 
Thus, demographic factors were not included as variables in the study. Because of the uniform 
backgrounds and entiy levels of the students, differences among them in course performance may 
depend on their engagement in the course. Table 2 summarizes course information for students 
included in this study. 

Table 2 


Blended Course Participation 


Course title 

Instructors 

Students 

Sections 

LSS 2533 Research Methods, 3 credits; this 
course is designed to introduce learners to 
the techniques and methods of research. 

Seven liberal studies 
specialists with 
graduate degrees in the 
field 

477 enrolled 
and 321 
completed 

Seven 


Instrument 

The success of students in blended learning courses can be measured by the grades students earn 
in the courses and academic performance on standardized exams (Ronsisvalle & Watkins, 2005). 
In the present study, student final scores in the course were determined by performance in a range 
of course assessments developed by expert panels for use across the national college system for 
these courses. Assessment methods were reflective of an undergraduate survey course, and 
included discussion, essays, limited-response exams, and projects. The assessments have been 
validated for their use over time and at multiple campuses. Student final course grade was used as 
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the measurement for success. Data logs generated by the Sakai CLE during the course included the 
number and length of student logins to the online course components. 

Data Collection 

The number of logins by each participant (a potential indicator of course presence during the 
weeks that the LMS was used) and average session length (a potential indicator of course 
engagement) were used as independent variables in our study. Login and final grade summary 
statistics are shown in Table 3 for the full 18-week semester of the course. 

Table 3 


Descriptive Statistics for Student Course Logins, Session Length, and Grades 


Variable 

Min 

25th % 

Median 

Mean 

75th % 

Max 

Number of logins per student 

1 

10 

15 

18.7 

24 

80 

Accumulated session time per student 
(hours) 

1.02 

11.37 

17.07 

24.35 

33.02 

98.70 

Average session length per student 
(hours) 

0.40 

1.07 

1.21 

1.26 

1.43 

3.06 

Course grade 
(percent) 

15.62 

70 

78 

74.76 

82 

100 


Note. Min = minimum; 25*% = lower quartile; 75 th % = upper quartile; Max = maximum 


Notice that while there was a wide range of login attempts (1-80), the interquartile range was 
much smaller with 50% of all students logging in between 10 and 24 times with a median of 15 
logins. Typical session lengths lasted about 1-1.5 hours. A positive correlation was found between 
number of logins and session length (r = 0.19 ,p = 0.0006; Figure 1). 
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Figure 1. Plot of average session length versus number of logins along with the corresponding 
regression line. 

Data Analysis 

We looked at two nested models for predicting final course grade based on using number of logins 
( Logins ) and average session length (ASL) as predictors. In Model 1, final course grade is linearly 
related to both predictive variables. In Model 2, we define the quadratic relationship: 


Model 2: Grade 

= bo+bi Logins + b 2 (Logins) 2 + b 3 ASL + b 4 (ASL) 2 + b 5 (ASL 
* Logins) 


Second order equations in the predictive variables were used to detect optimal performance at 
intermediate levels; for example, to allow us to determine if an intermediate number of logins 
predicts better performance than either too few or too many logins. All models were fit using the 
open-source software R. Due to the presence of extreme outliers (which could not be excluded on 
basis of incomplete information) and lack of normality in the distribution of residuals, the robust 
and resistant method of MM-Estimators, introduced by Yohai (1987) and implemented in R’s 
MASS package (Venables & Ripley, 2002), was used to obtain estimates and standard errors for 
linear model coefficients. Significance of parameters was determined using asymptotic t tests. 
Estimates for the optimal logins and ASL were obtained via standard formulas for the vertex of a 
paraboloid. The collinearity between logins and ASL was addressed by checking the variance 
inflation factors (VIF) associated with each term in Model 1. 
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Results 

Model 2 significantly outperformed Model l demonstrating a parabolic relationship between our 
response variable (course grade) and both predictors (logins and ASL) as shown in Table 4. Logins 
appears to be a more significant predictor than ASL although since these variables are collinear, 
we leave both terms in the model. Both quadratic coefficients in Model 2 are negative implying 
that course grade is predicted to be highest at intermediate values of both predictors, as shown in 
Figure 2. We approximate that the optimal combination of login times and session length was 
approximately 22 logins lasting around 63 minutes on average. 

Table 4 


Least Squares Estimates of Model Parameters 



Intercept 

Logins 

Logins A 2 

ASL ASL a 2 

Logins*ASL 

Model 1 

82.695*** 

-.070 


-.063* 


Model 2 

72.398*** 

•350 

-.012*** 

.112 -.001 

.003 


Note, ^denotes results which were significantly nonzero with p < 0.05; *** denotes results with p < 
0.001. To address the collinearity of predictors in Model 1, we computed the VIF of both terms and 
found a small impact on the resulting standard errors (VIF of Logins = 1.020; VIF of ASL = 1.020). 
Model 2 demonstrated a significant improvement over Model 1 (extra sum of squares F test; -F 3 , 3 2i 
= 5.466, p = 0.001). 
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Figure 2. Left or top panel: plot of course grade versus number of logins along with fit from Model 
2 using optimal ASL of 63.29 hours (solid black line). The optimal value of 22.33 logins is shown 
as the solid red line. Right or lower panel: plot of course grade versus ASL along with fit from 
Model 2 using optimal logins of 22.33 (solid black line). The optimal value of 63.29 hours is shown 
as the solid red line. 
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We propose possible explanations for our results. First, success at intermediate session lengths is 
aligned with common theories from learning sciences in that too much time spent per session may 
indicate cramming and lead to cognitive overload, while too little time may not contribute to 
sufficient time on task for retention (Cavanaugh, 2009; Liu & Cavanaugh, 2011). Since number of 
logins is positively correlated with ASL (Figure 1), this would also explain the dependence of 
course grade on number of logins. Alternatively, we conjecture that overly frequent logins may 
indicate a distracted student. Such a student may be logging into Sakai CLE frequently, but 
spending the bulk of their login time switching between the course Web site and non-course 
related Web browsing. As long as the student continued to stay logged in to Sakai, a long session 
length would have been recorded even though the student did not spend much of their login time 
on course related exercises. Such distracted sessions may, in turn, beget more distracted sessions, 
leading to a further increase in the frequency of logins. Finally, a third explanation is the classic 
correlation does not imply causation paradigm. It is possible that students who logged in too many 
times were less prepared students and the additional login time was not sufficient to overcome 
these initial aptitude gaps. Further study accounting for entiy level skills is needed to parse this 
possibility. 


Conclusions and Implications 

We examined student participation in the online components of a blended undergraduate college 
course in order to identify the extent to which the number and length of logins corresponded with 
student grades in the course. 

It was not the intent to compare the effects of learning in blended courses with other learning 
environments or to evaluate the specific design approach of the course. This study focused on 
student engagement in the online components of the course in order to guide students, instructors, 
and course designers. Each blended learning program is unique and designed for a specific 
purpose (Ferdig, Cavanaugh, & Freidhoff, 2012). This blended course was intended primarily to 
expand the learning time for native Arabic-speaking students who used the online tools and 
resources at the times of day and for the length of time they needed to enhance their learning, 
given the wide range of logins and session lengths recorded. For these students, who also met in 
class weekly, one to two logins per week of just over one hour in duration each related to the 
highest course grades. This result confirms the claim that variation theory underlies effective 
blended learning experiences because a range of activities maintains student engagement (Oliver 
& Trigwell, 2005). In keeping with variation theory and this study’s findings, effective blended 
learning courses can be designed to alternate physical and virtual experiences that require an hour 
of online time. 

Education in any learning environment is a complex, challenging professional practice; education 
across various learning environments multiplies the complexity and challenge. Formal research is 
needed in order to illuminate promising practices for each intersecting student and learning goal. 
This paper sheds light on one example. Data from practice must be coupled with data from 
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research. Data collection on learning processes and student outcomes is identified as a success 
factor in blended courses (Ferdig, Cavanaugh, & Freidhoff, 2015), and is a key step toward 
application of analytics to inform content selection, course design, and learning pathways 
(Cavanaugh, Sessums, & Drexler, 2015). Thus examining data on student learning activities, 
content usage, and performance over time with prediction and prescription stand to dramatically 
improve education attainment. 

We implore readers to do what they can to support research in blended education with open 
source tools: open your programs to researchers, engage in scholarship of teaching and learning 
(SoTL) to share your lessons, seek researcher funding and partnerships, disseminate research 
findings, participate in research conferences, read research journals, and assist researchers-in¬ 
training as mentors. We are all members of the community of practice and scholarship that 
advances outcomes for students as we learn together. 


References 

Bonk, C., & Graham, C. (2005). Handbook of blended learning: Global perspectives, local designs. San 

Francisco, CA: Pfeiffer. Retrieved from http: //www.wilev.com/WilevCDA/WilevTitle/productCd- 
Q 787 Q 77 f; 86 .html 

Cavanaugh, C. (2009). Getting students more learning time online: Distance education in support of 
expanded learning time. Washington, DC: Center for American Progress. Retrieved from 

https: / / www.americanprogress.org / wp- 
content/uploads/issues/20QQ/oF;/pdf/distancelearning.pdf 

Cavanaugh, C., & Hargis, J. (2010). Redefining school from site to service: Learning in and from K-12 

online education. Distance Learning, 7(2), 1-5. Retrieved from http://www.infoagepub.com/dl- 
issue.html?i=p 54 cn 6 bo 86 QiQ 

Cavanaugh, C., Sessums, C., & Drexler, W. (2015). A call to action for research in digital learning: 

Learning without limits of time, place, path, pace...or evidence. Journal of Online Learning 
Research, t(i) 9-15. Retrieved from https://www.editlib.org/p/i4Q8f;i/ 

Clark, T., & Barbour, M. (2015). Online, blended and distance education in schools. Sterling, VA: Stylus 
and Microsoft. Retrieved from 

https://stv.presswarehouse.com/Books/BookDetail.aspx?productID=3Qi638 

Ferdig, R., Cavanaugh, C., & Freidhoff, J. (Eds.). (2012). Lessons learned from blended programs: 
Experiences and recommendations from the field. Vienna, VA: iNACOL. Retrieved from 
http://www.inacol.org/resource/lessons-learned-from-blended-programs-experiences-and- 

recommendations-from-the-field/ 


273 












Participation in the Virtual Environment of Blended College Courses: An Activity Study of Student Performance 

Cavanaugh, Hargis, and Mayberry 

Ferdig, R., Cavanaugh, C., & Freidhoff, J. (2015). Research into K-12 online learning. In T. Clark, & M. 
Barbour (Eds.), Online and distance education in schools: Global perspectives on policy and 
practice. Sterling, VA: Stylus and Microsoft. 

https://stv.presswarehouse.com/Books/BookDetail.aspx?productID=3Qi638 

Ferdig, R., & Kennedy, K. (2014). Handbook of research on K-12 online and blended learning. Pittsburgh, 
PA: Carnegie Mellon University Press. Retrieved from 

http://press.etc.cmu.edu/content/handbook-research-k-i 2 -online-and-blended-learning-o 

Garrison, R., & Vaughan, H. (2008). Blended learning in higher education: Framework, principles and 
guidelines. San Francisco, CA: Jossey-Bass. 

http://www.wilev.com/WilevCDA/WilevTitle/productCd-0787Q87700.html 

Graham, L., Labonte, R., Roberts, V., O’Byrne, I., & Osterhout, C. (2014). Open learning in K-12 online 
and blended learning environments. In R. Ferdig & K. Kennedy (Eds.), Handbook of research on 
K—12 online and blended learning. Pittsburgh, PA: Carnegie Mellon University Press. Retrieved 
from http://press.etc.cmu.edu/content/handbook-research-k-12-online-and-blended-learning-o 

Keeler, C. (2015). Instructional design. In T. Clark & M. Barbour (Eds.), Online, blended and distance 
education in schools. Sterling, VA: Stylus and Microsoft. Retrieved from 

https://stv.presswarehouse.com/Books/BookDetail.aspx?productID=.2Qi638 

Khelifi, A., Abu Talib, M., A 1 Shabibi, A., A 1 Zaabi, A., & A 1 Marshoudi, N. (2010). Open source software in 
the UAE: A survey research. Paper presented at the 3rd International Symposium on Web 
Services, Zayed University, Dubai, U.A.E. Retrieved from 

http://brandengroup.com/files/OpenSource UAE.pdf 

Laumakis, M., Graham, C., & Dziuban, C. (2009). The Sloan-C pillars and boundary objects as a 

framework for evaluating blended learning. Journal for Asynchronous Learning Networks, 13(1). 
Retrieved from http://onbnelearningconsortium.org/read/iournal-issues/ 

Liu, F., & Cavanaugh, C. (2011). Online core course success factors in virtual school: Factors influencing 
student academic achievement. International Journal of E-Learning, 10(4), 43-65. Retrieved 
from https://www.editlib.org/p/.t.to^/ 

Oliver, M., & Trigwell, K. (2005). Can “blended learning” be redeemed? E-Learning and Digital Media, 
2(1), 17-26. doi:io.2304/elea.2005.2.i.2 

Picciano, A., & Dziuban, C. (Eds.). (2010). Blended learning: Research perspectives. Needham, MA: The 
Sloan Consortium. Retrieved from http: //ole.onlinelearningconsortium.org/node/ q21 

Ronsisvalle, T., & Watkins, R. (2005). Student success in online K-12 education. Quarterly Review of 
Distance Education, 6, 117-124. Retrieved from http://www.infoagepub.com/qrde- 
issue.html?i=pf;4C3cb86aQaii 


274 













Participation in the Virtual Environment of Blended College Courses: An Activity Study of Student Performance 

Cavanaugh, Hargis, and Mayberry 

U.S. Department of Education. (2009). Evaluation of evidence-based practices in online learning: A 
meta-analysis and review of online learning studies. Washington, DC: Office of Planning, 
Evaluation, and Policy Development. Retrieved from 

http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf 

Venables, W., & Ripley, B. (2002). Modern applied statistics with S (4 th ed.). New York, NY: Springer. 
Retrieved from http://hnk.springer.c0m/b00k/10.1007%2Fq78-0-387-21706-2 

Yohai, V. (1987). High breakdown point and high efficiency robust estimates for regression. The Annals of 
Statistics, 15(20), 642-656. Retrieved from http://www.istor.org/stable/22413.21 


D Athabasca 
University 



275 






